Image pickup system for reproducing image data using sensitivity function

ABSTRACT

In a sensitivity function measuring unit, a CMD for picking up a beam from a light source is arranged through a photographic lens and a liquid crystal lens. A preamplifier, an A/D converter, a subtractor, a sensitivity function memory, a compressor, and R, G, and B sensitivity function memories are sequentially connected to the CMD through a switching circuit. An image reproducing unit for reproducing an original image using a sensitivity function calculated by the sensitivity function measuring unit is connected to a calculator for calculating a reproduced image, a frame memory for storing an image signal picked up by the CMD, and calculators for calculating sensitivity functions stored in the sensitivity function memories. The image reproducing unit is also connected to a display unit.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention generally relates to an image pickup system and,more particularly, to an image pickup system having an image reproducingmeans for correctly reproducing degraded image data in, e.g., an imagepickup system such as an electronic still camera for picking an originalimage to reproduce the original image.

Description of the Related Art

As is known, in an imaging system such as an electronic still camerausing an optical system, as shown in FIG. 3A, an original image f(r) (rrepresents a position) is formed on an image pickup element (not shown)through an optical system L as an observation image g(r).

In this case, when the Fourier spectrum of the original image f(r) isrepresented by F(ω), and the Fourier spectrum of the observation imageg(r) is represented by G(ω), the following equation can be obtained:

    G(ω)=H(ω)×F(ω)                     (1)

(where ω: spatial frequency)

In equation (1), H(ω) is called an OTF (Optical Transfer Function), andis used for representing image forming characteristics of an imagingsystem.

In addition, when H(ω) is subjected to inverse Fourier transformation, aPSF (Point Spread Function) is obtained.

In order to cause the observation image g(r) to coincide with theoriginal image f(r), H(ω)=1 must be satisfied for all spatialfrequencies ω.

However, in a practical optical system, H(ω)<1 is satisfied, and adegraded image is formed.

A method using an inverse filter is known as a method of reproducing anoriginal image from an observation image.

The inverse filter is described in detail in, e.g., "Fundamentals ofDigital Image Processing", ANIL K. JAIN, pp. 275 to 277, Prentice-HallInternational Editions.

According to this literature, as a reproducing filter, the followingequation is given:

    H.sup.- (ω)=1/H(ω)                             (2)

However, since this filter is represented by the reciprocal number ofH(ω), when H(ω)=0 is satisfied, H⁻ (ω) is diverged. Therefore, thefollowing equations are defined:

    H.sup.- (ω)=1/H(ω) (when H(ω)≠0)   (3)

    H.sup.- (ω)=0 (when H(ω)=0)                    (3')

In a reproducing filter represented by equations (2), (3), and (3'), anintensity distribution of the image of an optical system must beuniformed at any position of the image, i.e., PSFs in the image must beequal to each other (space-invariant).

However, in a practical optical system, since the PSFs are changed inaccordance with their positions due to various aberrations, failure infocusing, or the like, the original image cannot be correctly reproducedby the reproducing filter represented by equations (2) and (3).

In this case, after the PSFs which are changed in accordance with theirpositions are correctly measured, the reproducing filter represented byequations (2) and (3) may be used. However, sampling of the PSFsperformed prior to the measuring of the PSFs is posed as a problem.

That is, although an image on an observation image plane is defined as acontinuous image, the image is separated into pixels in an image pickupelement or the like, and the image is discretely sampled. Therefore, thePSFs cannot be correctly measured.

For this reason, it is essentially impossible to correctly reproduce anoriginal image by a conventional reproducing filter.

SUMMARY OF THE INVENTION

It is, therefore, an object of the present invention to provide a newand improved image pickup system which can correctly reproduce anoriginal image without being influenced by an optical system even when acontinuous system and a discrete system are used.

According to the present invention, there is provided an image pickupapparatus comprising:

a photographic optical system directed to an object to be photographed;

image pickup means for outputting image data corresponding to an imageof the object which is incident through the photographic optical system;

sensitivity function storing means for prestoring sensitivity functiondata representing a light sensitivity in an object space at each pexelincluded in the image pickup means; and

image reproducing means for reproducing the image data output from theimage pickup means using the sensitivity function data stored in thesensitivity function storing means.

Additional objects and advantages of the invention will be set fourth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention ma be realized and obtained by means ofthe instrumentalities and combinations particularly pointed out in theappended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate presently preferred embodiments ofthe invention and, together with the general description given above andthe detailed description of the preferred embodiments given below, serveto explain the principles of the invention.

FIG. 1 is a view showing the arrangement of a sensitivity functionmeasuring means used in the first embodiment of the present invention

FIG. 2 is a view showing the arrangement of an image reproducing meansaccording to the first embodiment of the present invention;

FIGS. 3A and 3B are views for explaining sensitivity functions of thepresent invention;

FIG. 4 shows sensitivity functions of the present invention;

FIGS. 5A and 5B are views showing a field angle setting mechanismaccording to the present invention;

FIG. 6 is a view showing the arrangement of a modification of thesensitivity function measuring means according to the first embodimentof the present invention;

FIGS. 7A and 7B are views showing modifications of the image settingmechanism according to the first embodiment of the present invention;

FIG. 8 is a view showing an arrangement for forming sensitivityfunctions and a reproducing filter in the second embodiment of thepresent invention;

FIG. 9A is a view showing a color filter array used in the secondembodiment of the present invention;

FIG. 9B is a view showing the arrangement of a reproducing filter usedin the second embodiment;

FIG. 10 is a view showing the arrangement of an image reproducingcircuit according to the second embodiment of the present invention;

FIG. 11 is a view showing the arrangement of a reproduction processingcircuit according to the third embodiment of the present invention;

FIG. 12A is a view showing the arrangement of a distance detecting unitused in the third embodiment of the present invention;

FIG. 12B is a graph for explaining that distance data capable ofobtaining a maximum contrast value is output from the distance detectingunit used in the third embodiment of the present invention;

FIGS. 13A and 13B are views for illustrating the shapes of dividedscreens according to the fourth embodiment of the present invention;

FIG. 14 is a view showing the arrangement of a main part of areproducing filter calculating circuit for each divided area; and

FIG. 15 is a view showing the arrangement of a reproduction processingcircuit according to the fourth embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the presently preferredembodiments of the invention as illustrated in the accompanyingdrawings, in which like reference characters designate like orcorresponding parts throughout the several drawings.

Prior to a description of the embodiments of the present invention, amethod of reproducing an original image using sensitivity functionsserving as the principle of the present invention will be describedbelow.

An original image f(r) which is a continuous image shown in FIG. 3A isformed by an optical system L, and an observation image g(r) serving asa discrete image is picked up by an image pickup element (not shown)placed on the image formation plane. In this case, a sensitivityfunction storing means and an image reproducing means for reproducingthe original image by a sensitivity function stored in the sensitivityfunction storing means will be considered.

In an image pickup operation by the image pickup element, as shown inFIG. 3B, the observation image g(r) is regarded as a discrete image andcan be modeled as the following equation:

    gi=∫f(r)×hi(r)dr                                (4)

In equation (4), hi(r) is a function representing a sensitivitydistribution contributing to the ith pixel of the observation imageg(r), and all pixels are expressed as the following equation:

    g(r)=H{f(r)}                                               (5)

In equation (5), H is regarded as a transformation operator from acontinuous image to a discrete image, and is called a sensitivityfunction.

The original image f(r) serving as a continuous image is defined by acontinuous system, so that an infinite number of points are present inthe range of the observation image. Therefore, an infinite number ofbasic vectors in an object space are given.

For this reason, the number of columns of the sensitivity function H isinfinite, and the sensitivity function itself cannot be defined.

A transposed matrix H^(T) of H is defined as a transformation operatorfrom a discrete system to a continuous system, and HH^(T) is considered.That is, assuming that the number of pixels of the image pickup elementis, e.g., m, HH^(T) is defined as an m×m matrix, an inverse matrix canbe obtained. A reproduced image fe(r) can be obtained by the followingequation:

    fe(r)=H.sub.S.sup.T (HH.sup.T).sup.+ g(r)                  (6)

In equation (6), + represents pseudo inverse, i.e., a pseudo inversematrix.

In addition, when each element of HH^(T) is represented by aij, thefollowing equation is obtained:

    aij=∫hi(r)×hj(r)dr                              (7)

(HH^(T))⁺ can be calculated by various methods. For example, (HH^(T))can be calculated by an SVD analysis.

The SVD stands for Singular Value Decomposition. The SVD is used incompression of image data and in an analysis of characteristics of alinear system, and is described in the above literature, pp. 176 to 180.

When noise is generated, HH^(T) +cI (c is a constant, and I is a unitmatrix) may be used in place of HH^(T) in equation (6).

In addition, since a position r of an original image can be arbitrarilyset in equation (6), it means that an image signal at any position onthe original image can be obtained independently of the pixel count m ofthe image pickup element.

That is, as an image in an arbitrary area can be picked up in accordancewith the pixel count m, electronic zooming can be performed withoutperforming interpolatory calculation.

In this case, H_(S) ^(T) can be understood as transformation forrepresenting a reproduced image, the suffix s represents the number ofdisplay pixels, and H_(S) ^(T) is represented by an s×m matrix.

As described above, the sensitivity functions H are measured by asensitivity function measuring means in place of measurement of PSFs incalculation, and an observation image defined as a discrete image isreproduced as a continuous image continuous like an original image by animage reproducing means using equation (6).

The sensitivity function measuring means will be described withreference to FIG. 1 showing an arrangement thereof. Although atwo-dimensional image is processed in practice, the image is representedas an one-dimensional image for descriptive convenience.

That is, in FIG. 1, an illumination light source 1 is arranged such thata beam emitted from a point light source is picked up by a CMD 10through a field stop 3 and a photographic lens 4.

The illumination light source 1 is fixed on an X-Y stage 2 moved withina plane perpendicular to the optical axis of the photographic lens 4.

In order to change the optical path of the beam passing through thephotographic lens 4, a liquid crystal lens 5 is vertically arranged tothe photographic lens 4.

In this case, the field stop 3 and the liquid crystal lens 5 areconnected to a field angle setting circuit 21 for driving a stop driver6 and a liquid crystal lens driver 7 for controlling the field stop 3and the liquid crystal lens 5, respectively.

The beam passing through the liquid crystal lens 5 is focused on thecharge modulation device (to be referred to as a CMD hereinafter) 10serving as an image pickup element through a rotary color filter 8connected to a color filter driver 9.

A preamplifier 12 and an analog/digital (A/D) converter 13 are connectedto the CMD 10.

In addition, the A/D converter 13 and a memory (to be referred to as anFPN memory hereinafter) 15 in which the fixed pattern noise of the CMD10 is stored in advance are connected to the two input terminals of asubtractor 14, respectively.

A memory 16 (to be referred to as a sensitivity function memoryhereinafter, because the memory 16 is to store especially sensitivityfunctions) for storing data obtained by an image pickup operation isconnected to the output terminal of the subtractor 14.

In this case, although the number of columns of sensitivity function His originally infinite, measurement of the sensitivity function H cannotbe impossible, so that the number of the columns is set to be finite(n), and the sensitivity functions are stored in the memory in the formof an m×n matrix as shown in FIG. 4.

Each of the rows stores a sensitivity distribution hi(r) of acorresponding one of pixels i.

A switching circuit 18 for performing a switching operation between ared (R) sensitivity function memory 19r, a green (G) sensitivityfunction memory 19g, and a (B) blue sensitivity function memory 19b forthree primary colors (RGB) is connected to the output terminal of thesensitivity function memory 16 through a data compressor 17.

An address controller 20 is connected to the R sensitivity functionmemory 19r, the G sensitivity function memory 19g, and the B sensitivityfunction memory 19b.

In addition, the X-Y stage 2, the field angle setting circuit 21, thecolor filter driver 9 for controlling the rotary color filter 8, the CMDdriver 11 for controlling the CMD 10, the FPN memory 15, the switchingcircuit 18, and the address controller 20 are connected to thecontroller 22.

The measurement of the sensitivity function H on the basis of the abovearrangement will be described below.

The measurement is performed for each of R, G, and B color images inunits of field angles.

In this case, as described above, the sensitivity function H is antransformation operator from a continuous system to a discrete system,and the number of columns of the sensitivity function H is infinite. Themeasurement is, however, impossible in practice. Assuming that thenumber of columns of the sensitivity function H is set to be n, themeasurement is performed, and necessary positions are calculated byinterpolatory calculation as will be described later.

In the sensitivity function H, as shown in FIG. 4, a sensitivitydistribution of each observation image position is arranged in a rowdirection, and a sensitivity distribution of each observation imageposition is arranged in a column direction.

The sensitivity function is obtained by performing an image pickupoperation at each observation image position while the illuminationlight source 1 is shifted.

In order to perform a practical operation of a sensitivity functionmeasuring means, after the rotary color filter 8 is set to be R by thecolor filter driver 9, the field stop 3 is adjusted under the control ofthe stop driver 6 such that a field angle is set to be θmax(corresponding to FIG. 5A) by the field angle setting circuit 21, andthe liquid crystal lens 5 is adjusted under the control of the liquidcrystal lens driver 7.

The entire field of the original image is equally divided by n positions(x1, x2, . . . , xn), and the illumination light source 1 is moved tothe position x1 on the original image by driving the X-Y stage 2 toperform an image pickup operation by the CMD 10.

Setting of a field angle performed by the field stop 3 and the liquidcrystal lens 5 will be described below with reference to FIGS. 5A and5B.

According to the present invention, as described above, electroniczooming can be performed without specially changing a field angleaccording to equation (6). However, the accuracy of reproduction can beimproved by inputting only the beam of an image to be photographed andto be reproduced as much as possible. For this purpose, the field anglesetting circuit 21 is arranged.

More specifically, FIG. 5A shows a case corresponding to a wide-anglelens. In this case, the field stop 3 is widely open, and the refractiveindex of the liquid crystal lens 5 is controlled to increase arefracting power.

In contrast to this, FIG. 5B shows a case corresponding to a telephotolens. In this case, the field stop 3 is narrow, and the refractive indexof the liquid crystal lens 5 is controlled to decrease a refractingpower.

That is, in a telephoto mode, when the refractive index of the liquidcrystal lens 5 is equal to that of the liquid crystal lens 5 in FIG. 5A,the beam of an image to be photographed is incident on only a part ofthe image pickup element. However, when the refractive index isdecreased as shown in FIG. 5B, the beam of the image can be incident onthe entire image pickup element without any waste.

In addition, a field angle can easily be changed by using the field stop3 and the liquid crystal lens 5.

The field angle can be changed every predetermined field angle (to bereferred to as iint hereinafter) within a range of θmin to θmax whichcan be set by the field angle setting circuit 21 through the controller22.

An image signal obtained by picking up the image of the pixel i of theCMD 10 is a sensitivity distribution hi(x1), the image signal isamplified by the preamplifier 12, converted into a digital signal by theA/D converter 13, and then written in the first column of each row ofthe sensitivity function memory 16 as a sensitivity distribution signal,i.e., sensitivity function data, obtained by causing the subtractor 14to subtract the fixed pattern noise of the CMD stored in the FPN memory15.

The illumination light source 1 is moved to the position x2, the sameprocessing as described above is performed, and the resultantsensitivity function data is written at a position corresponding to thesecond column of each row of the sensitivity function memory 6.

The same processing as described above is repeated until theillumination light source 1 reaches the position xn, sensitivityfunction data of each position is written in a corresponding one ofcolumns of the sensitivity function memory 16.

The amount of each data of the sensitivity function memory 16 iscompressed by the data compressor 17, and the compressed data is storedin the R sensitivity function memory 19r as sensitivity function dataHimax.

The field angle is decreased by iint, and the same processing asdescribed above is performed to obtain sensitivity function dataHimax-int. This sensitivity function data is stored in the R sensitivityfunction memory 19r.

Sensitivity function data of all the fields having field angles whichare different from each other by iint are stored, and the sameprocessing as described above is repeated until the field angle is setto be imin finally.

Note that, in detection, an image pickup operation of the field isperformed at the n positions equally dividing the original image withinthe range of predetermined positions xl to xn.

In this manner, all the sensitivity function data for an R image arestored, and sensitivity function data for G and B images can be obtainedby the same manner as described above. The sensitivity function data arestored in the G sensitivity function memory 19g and the B sensitivityfunction memory 19b, respectively, thereby completely storing thesensitivity function data.

An image pickup apparatus including an image reproducing means forreproducing an image from the image pickup element by using thesensitivity function data obtained by the measurement will be describedbelow.

FIG. 2 is a view showing an arrangement of an image pickup apparatusaccording to the first embodiment. The image pickup apparatus comprisesan image reproducing means in which a method of reproducing an image isapplied to so-called zooming by using the sensitivity function dataobtained by the sensitivity function measuring means.

The same reference numerals as in FIG. 1 denote the same parts in FIG.2, and a detailed description thereof will be omitted.

In FIG. 2, a frame memory 23 for storing image data is connected to thesubtractor 14 of the sensitivity function measuring means in FIG. 1.

The R sensitivity function memory 19r, the G sensitivity function memory19g, and the B sensitivity function memory 19b are connected to the datacompressor 17 through the switching circuit 18 in FIG. 1. Thesesensitivity function memories 19r, 19g, and 19b are connected to onlythe address controller 20 in FIG. 2.

As described above, the sensitivity function data of the R, G, and Bimages are stored in the memories 19r, 19g, and 19b, respectively, and aswitching circuit 24 for selecting the outputs of the memories 19r, 19g,and 19b is connected to only the controller 22.

The switching circuit 24 serving as an image reproducing means isconnected to an (HH^(T))⁺ calculator 26 and an H_(S) ^(T) calculator 27through a data expander 25.

In addition, in order to calculate reproduced image data by the imagedata and calculation outputs from the calculators, the calculators 26and 27 and the frame memory 23 are connected to an fe(r) calculator 28.

A display unit 29 for displaying the reproduced image of an object 30 tobe photographed which is present on the front surface of the liquidcrystal lens 5 through the field stop 3 is connected to the fe(r)calculator 28.

An operation of the image reproducing apparatus on the basis of theabove arrangement will be described below.

When the image of the object 30 begins to be picked up, the field stop 3and the liquid crystal lens 5 are adjusted by the field angle settingcircuit 21 through the controller 22 such that a desired field angle θis obtained.

An exposure period of the CMD 10 is set by a photometric system (notshown), and the rotary filter 8 is set to be R by the color filterdriver 9 through the controller 22.

An image signal obtained by causing the CMD 10 to pick up the object 30through the optical systems 3, 4, and 5 is amplified by the preamplifier12, converted into a digital signal by the A/D converter 13, and storedin the frame memory 23 as image data g(r) after an FPN is subtracted bythe subtractor 14.

Sensitivity function data H corresponding to the field angle θ is readout from the R sensitivity function memory 19r by the address controller20, and is input to the data expander 25 through the switching circuit24. The sensitivity function data H is decoded by the data expander 25,and is input to the (HH^(T))⁺ calculator 26 and the H_(S) ^(T)calculator 27.

Each component of HH^(T) is calculated in the (HH^(T))⁺ calculator 26according to equation (7), and (HH^(T))⁺ is then calculated.

In the H_(S) ^(T) calculator 27, s elements for display are calculatedfrom a sensitivity distribution hi(r) consisting of n elements byinterpolatory calculation. These s elements are transposed to obtainH_(S) ^(T), and H_(S) ^(T) is input to the fe(r) calculator 28.

In the f(r) calculator 28, the reproduced image data fe(r) of an R imageis calculated on the basis of g(r) input from the frame memory 23,(HH^(T))⁺ input from the (HH^(T))⁺ calculator 26, and H_(S) ^(T) inputfrom the H_(S) ^(T) calculator 27, and the reproduced image data fe(r)is output to the display unit 29.

After the reproduced image data of G and B images are calculated in thesame manner as described above, the reproduced image data are output tothe display unit 29 to be displayed.

When the field angle is set to be a different value, sensitivityfunction data corresponding to the field angle is read out from each ofthe sensitivity function memories 19r, 19g, and 19b, and the sameprocessing as described above is performed.

As described above, according to this embodiment, when a photographiclens has aberrations or focusing errors, an original image can bereproduced by a sensitivity function without measuring a PSF. For thisreason, the present invention can be utilized in automatic focusingwithout using a large number of lenses.

At this time, the liquid crystal lens is used as part of the opticalsystem, and the optical path of the liquid crystal lens is changed suchthat the beam of the object to be photographed is incident on the entireimage pickup element without any waste. In addition, when the field stopis driven in accordance with the change in optical path, the accuracy ofa reproduced image can be improved.

In this embodiment, since sensitivity function data is compressed andstored, a storage capacity for the sensitivity function data can bedecreased.

In addition, in this embodiment, since an original image is reproducedby using sensitivity function data of the three primary colors, i.e.,RGB, a color image can be properly reproduced.

In this embodiment, the number of pixels of a displayed image can beeasily changed by changing H_(S) ^(T).

In this embodiment, even when the image pickup element has defectivepixels, an original image can be correctly reproduced.

In this embodiment, an image is reproduced by using all the pixels ofthe CMD 10. According to the present invention, when the sensitivityfunction data H is obtained, the number of pixels of the image pickupelement or the positions of the pixels are not limited. An image can becorrectly reproduced by, e.g., random sampling using an arbitrary numberof pixels.

In this embodiment, although the sensitivity function data H is stored,H_(S) ^(T) (HH^(T))⁺ may be calculated during storage of the sensitivityfunction data and stored.

In this embodiment, the illumination light source is driven in measuringa sensitivity function. However, the illumination light source may befixed, and the image pickup unit constituted by a CMD and lens systemsmay be driven.

In addition, in the sensitivity function storing means, as shown in FIG.6, a point corresponding to the illumination light source may besequentially moved and displayed by a monitor 31 and a displaycontroller 32 to measure the sensitivity function.

Although an optical path is changed using the liquid crystal lens inzooming, as shown in FIGS. 7A and 7B, a plurality of lenses 33a and 33bmay be moved.

FIGS. 7A and 7B show a wide-angle image pickup mode and a telephotoimage pickup mode, respectively, and have a relationship correspondingto the relationship between FIGS. 5A and 5B.

Several embodiments different from the first embodiment will bedescribed below.

In the above first embodiment, as shown in FIG. 3, the measurement ofdata related to sensitivity and the reproduction of a picked image areperformed under the condition that a distance between an original imageand a photographic lens of an optical system is kept constant.

In practice, the sensitivity function is, however, changed in accordancewith the distance between the photographic lens and the original image.Therefore, the sensitivity function in accordance with the distance mustbe measured, and the original image must be reproduced on the basis ofthe sensitivity function.

The second embodiment on the basis of the above principle will bedescribed below with reference to FIGS. 8 to 10.

Note that the same reference numerals as in the first embodiment denotethe same parts in the second embodiment.

FIG. 8 is a view showing an arrangement for forming sensitivityfunctions and reproducing filters according to the second embodiment ofthe present invention.

The embodiment in FIG. 8 is different from the embodiment shown in FIG.1 in the following points. That is, a stripe type color filter array 39shown in FIG. 9A is used in place of a rotary color filter to obtain acolor signal, and reproducing filters are formed in accordance with adistance Di between a illumination light source 1 and a photographiclens 4.

In FIG. 8, reference numeral 34 denotes a distance measuring unit (AFsensor), and the distance measuring unit 34 measures the distance Di.

Reference numeral 35 denotes a color separation circuit. In the colorseparation circuit 35, a color signal obtained by an image pickupoperation using the color filter array 39 is separated into R, G, and Bcolor signals, and the R, G, and B signals are output to R, G, and Bsensitivity function memories 16r, 16g, and 16b, respectively.

Reference numerals 36r, 36g, and 36b denote reproducing filtercalculators. As shown in FIG. 9B, each of the reproducing filtercalculators is constituted by an (HH^(T))⁺ calculator 26, an H_(S) ^(T)calculator 27, and a multiplier 37, thereby calculating a reproducingfilter, i.e., H_(S) ^(T) (HH^(T))⁺.

Reference numerals 38r, 38g, and 38b denote memories for storing thereproducing filters calculated for R, G, and B data, respectively. Eachof the memories is arranged such that the reproducing filter is writtenat an address corresponding to the distance Di. The distance between theillumination light source and the photographic lens 4 is changed by anX-Y stage 2, the distance Di is measured by the distance measuring unit34 (AF sensor), and each calculated reproducing filter is written in acorresponding one of the reproducing filter memories. The distance Di isselected from distances D₁ to D_(I) ranging from, e.g., a closestfocusing distance to a distance (x).

FIG. 10 shows an image reproducing circuit.

In FIG. 10, reference numerals 40r, 40g, and 40b denote filtering units,and the filtering units 40r, 40g, and 40b calculate reproduced imagedata R', G', and B' on the basis of image data g_(R), g_(G), and g_(B)from the R, G, and B frame memories 16r, 16g, and 16b in accordance withthe following equations:

    R'=e.sub.R g.sub.R                                         (8)

    G'=e.sub.G g.sub.G                                         (8')

    B'=e.sub.B g.sub.B                                         (8")

In equations (8), (8'), and (8"), e_(R), e_(G), and e_(B) are n×mmatrices (where, n is the number of measured sensitivity functions, andm is the number of pixels of a CMD), and g_(R), g_(G), and g_(B) are m×1matrices.

In FIG. 10, reference numeral 41 denotes an output unit such as a CRT ora video printer.

An operation of reproduction processing will be described below.

After the image signal of an object 30 to be photographed picked up by aCMD 10 in an appropriate exposure amount is converted into a digitalsignal by an A/D converter 13 through a preamplifier 12, and the imagesignal is separated into R, G, and B data by a color separation circuit35. The R, G, and B data are stored in the frame memories 16r, 16g, and16b, respectively.

A distance Di between a photographic lens 4 and the object 30 ismeasured by the operation of a distance measuring unit 34, and anaddress controller 20 reads out reproducing filters e_(R), e_(G), ande_(B) corresponding to the distance Di from reproducing filter memories38r, 38g, and 38b, respectively.

The reproducing signals R', G', and B' obtained by causing the filteringunits 40r, 40g, and 40b to calculate equations (8), (8') and (8") areoutput to the output unit 41, thereby displaying an image on a CRT orthe like.

According to the second embodiment arranged as described above,reproducing filters are designed for each distance Di, and reproducingfilters are selected in accordance with the distance between the objectand the photographic lens during reproduction processing. For thisreason, even when the distance between the object and the photographiclens is changed, appropriate reproduction processing can be performed.

In addition, in the second embodiment, the color filter array is used inplace of a rotary color filter. A rotary color filter and a driving unittherefor are not required.

The color filter array is not limited to a stripe type color filterarray, and an appropriate color filter array such as a mosaic type orcheck type color filter array may be used.

According to the second embodiment, although a field angle is notchanged, the field angle may be changed as in the first embodiment.

In the second embodiment, the distance measuring unit (AF sensor) formeasuring the distance Di between the object and the photographic lensduring reproduction processing is required.

In the third embodiment, as will be described later, a distancemeasuring unit (AF sensor) is not used to decrease the size and weightof an image pickup unit.

FIG. 11 is a view showing the arrangement of a reproduction processingcircuit according to the third embodiment.

In FIG. 11, reference numeral 45 denotes a distance measuring unit. Asshown in FIG. 12, the distance detector 45 is constituted by a luminancedetector 42 for detecting luminance data Y' from reproducing data R',G', and B', a contrast detector 43 for detecting the contrast of animage from the luminance data Y' by using a band-pass filter or thelike, a contrast memory 46 for storing a contrast value from thecontrast detector 43, and a maximum value detector 44 for detecting themaximum value of the contrast value.

An operation of the third embodiment will be described below.

A signal obtained by picking up the image of an object to bephotographed as in the second embodiment is subjected to colorseparation, and the separated data are stored in frame memories 16r,16g, and 16b, respectively.

Reproducing filters corresponding to a distance D₁ between an object anda photographic lens are read out from reproducing filter memories 38r,38g, and 38b by an address controller 20 through a controller 22, andreproducing signals R', G', and B' are obtained by filtering units 40r,40g, and 40b, respectively.

Luminance data Y' is obtained by the luminance detector 42, and acontrast value is obtained by the contrast detector 43. The contrastvalue is written at a position corresponding to the distance D₁ in thecontrast memory 46.

Reproducing filters corresponding to a distance D₂ between the objectand the photographic lens are read out from the reproducing filtermemories, and a contrast value is obtained in the same manner asdescribed above. The value is written at a predetermined position.

The above processing is repeated up to a distance D_(I) between theobject and the photographic lens, and a contrast value corresponding toeach distance is obtained.

In the maximum value detector 44, a distance Dmax (FIG. 12B) having amaximum contrast value is output to the controller 22. Reproducingfilters corresponding to the distance Dmax are read out from thereproducing filter memories 38r, 38g, and 38b, and reproductionprocessing is performed by each of the filtering units 40r, 40g, and40b, thereby outputting reproduced data R', G', and B' to an output unit41.

As described above, according to the third embodiment, reproducingfilters each having the maximum contrast value are selected, andreproduction processing is performed. For this reason, an in-focus imagecan be obtained independently of the distance between the object and thephotographic lens

According to the third embodiment, a distance measuring unit (AF sensor)is not required, and the photographic lens need not be moved. Therefore,a lightweight, compact image pickup unit can be provided.

According to the third embodiment, in detection of contrast by thecontrast detector 43, the range and position of the contrast can befreely set, so that an in-focus position can be designated after aphotographic operation is performed.

In the third embodiment, reproducing filters are selected in an order ofdistances under the control of the controller. However, for example, thefollowing arrangement may be used. That is, a knob (not shown) forchanging a read position of a reproducing filter memory may be arranged,and an observer may operate this knob to select an image to be outputwhile observing a reproduced image on the screen of the output unit 41.

In the above embodiment, an image on the whole screen is reproduced, andan m×m generalized inverse matrix is calculated by an (HH^(T))⁺calculator.

Since m is generally about 500² =250,000, calculation is considerablycomplicated to obtain the generalized inverse matrix of HH^(T). For thisreason, a screen may be divided into a large number of areas, areproducing filter may be designed for each of the divided areas, andthe number of degrees of HH^(T) may be decreased. In this manner, thegeneralized inverse matrix can easily be obtained.

In the fourth embodiment, a case wherein a method of dividing a screenis employed will be described below.

In this case, as shown in FIG. 13A, the screen is divided into 4×4=16areas.

FIG. 14 is a view showing the arrangement of a main part of areproducing filter calculating circuit for each of the divided areasaccording to the fourth embodiment.

In FIG. 14, reference numeral 50 denotes a controller for write/readaccess to memories. In the fourth embodiment, sensitivity functions aremeasured in the same manner as in each of the above embodiments.

In reproducing filter calculators 47r, 47g, and 47b, reproducing filtersare calculated by using a sensitivity function corresponding to each ofthe divided areas. The reproducing filters are written in dividedreproducing filter memories 48r, 48g, and 48b.

FIG. 15 is a view showing the arrangement of a reproduction processingcircuit according to the fourth embodiment.

As the characteristic feature of the fourth embodiment, dividedreproducing filters corresponding to each of the divided areas are readout from the divided reproducing filter memories 48r, 48g, and 48r,reproduction processing is performed by filtering units 40r, 40g, and40b, and the divided area images which are reproduced are synthesized byimage synthesizing circuits 49r, 49g, and 49b, thereby reproducing theimage of the whole screen.

According to the fourth embodiment, as the number of divided areas isdecreased, the number of degrees of the matrix of HH^(T) is decreased.For this reason, the matrix can easily be calculated, and thecalculation period of reproducing filters can be shortened.

When the screen is divided into divided areas such that each of thedivided areas has an overlapping area therebetween as shown in FIG. 13B,synthesized images can be prevented from being discontinuously connectedto each other.

In addition, when a conventional sequential solution is used in place ofthe calculation of an inverse matrix, the above effect as describedabove can be obtained.

As described above, according to the present invention, sensitivityfunction data of each position of an image pickup element is properlystored, and an original image can be analytically, correctly reproducedas a continuous image by using the sensitivity function data of eachposition of the image pickup element with out being influenced by anoptical system.

Additional embodiments of the present invention will be apparent tothose skilled in the art from consideration of the specification andpractice of the present invention disclosed herein. It is intended thatthe specification and examples be considered as exemplary only, with thetrue scope of the present invention being indicated by the followingclaims.

What is claimed is:
 1. An image pickup apparatus comprising:aphotographic optical system directed to an object to be photographed;image pickup means for outputting image data corresponding to an imageof said object which is incident through said photographic opticalsystem; sensitivity function storing means for prestoring sensitivityfunction data representing a light sensitivity in an object space ateach pixel included in said image pickup means; and image reproducingmeans for reproducing the image data output from said image pickup meansby using the sensitivity function data stored in said sensitivityfunction storing means.
 2. An apparatus according to claim 1, whereinsaid sensitivity function storing means includes a red sensitivityfunction memory, a green sensitivity function memory, and a bluesensitivity function memory for three primary colors.
 3. An apparatusaccording to claim 1, wherein said photographic optical system includesfield angle setting means for a zooming application.
 4. An apparatusaccording to claim 3, wherein said field angle setting means includes aliquid crystal lens having a refractive index controlled in accordancewith a set field angle to change an optical path.
 5. An apparatusaccording to claim 3, wherein said field angle setting means includes aplurality of lenses moved in accordance with a set field angle to changean optical path.